System and method for employing model repository

ABSTRACT

Model metadata for each of a plurality of models is stored. The model metadata includes a statistical analysis technique identifier and one or more model input data identifiers. A request to execute a model is received. The request includes data identifying one of the plurality of models, and a model execution start date and end date. On the model execution start date, execution of the model associated with the model execution request is commenced. Outputs of the executed model are stored in a database. The outputs are associated with a model instance identifier, information describing a context for execution of the model, and model output type information. The outputs are retrieved, using the model instance identifier, for analysis.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/702,849, filed Sep. 19, 2012, which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This invention relates to use of repositories to store metadataassociated with mathematical models and outputs thereof.

SUMMARY

The present invention relates to a computer implemented method, a systemand a computer readable medium storing instructions which, when executedby a computer cause the computer to perform the described method. Modelmetadata for each of a plurality of models is stored. The model metadataincludes a statistical analysis technique identifier and one or moremodel input data identifiers. A request to execute a model is received.The request includes data identifying one of the plurality of models,and a model execution start date and end date. On the model executionstart date, execution of the model associated with the model executionrequest is commenced. Outputs of the executed model are stored in adatabase. The outputs are associated with a model instance identifier,information describing a context for execution of the model, and modeloutput type information. The outputs are retrieved, using the modelinstance identifier, for analysis.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofvarious embodiments, will be better understood when read in conjunctionwith the appended drawings. It should be understood, however, that theinvention is not limited to the precise arrangements andinstrumentalities shown.

FIG. 1 illustrates the exemplary context in which an embodiment of theinvention may be used;

FIG. 2 illustrates the components of a production model executionenvironment in accordance with one embodiment of the invention;

FIG. 3A is a diagram of an exemplary scheduler in accordance with anembodiment of the present invention;

FIG. 3B is a diagram of exemplary components of the model storage systemin accordance with an embodiment of the present invention;

FIG. 3C illustrates an exemplary flow of the proposed modelingenvironment, which uses the components shown in FIG. 3A and FIG. 3B, inaccordance with an embodiment of the present invention;

FIG. 3D is a flowchart illustrating the process outlined in FIG. 3C;

FIG. 4A is an exemplary entity-relationship model of statistical modelsand their executions, that may be used in connection with an embodimentof the present invention;

FIG. 4B is an exemplary entity-relationship model describing how thestatistical model relates to data within a database, that may be used inconnection with an embodiment of the present invention;

FIG. 5A is a logical data model that may be used in connection with anembodiment of the present invention;

FIG. 5B is a logical data model that may be used in connection with anembodiment of the present invention;

FIGS. 6A and 6B together show an exemplary implementation of a datamodel that may be used in connection with an embodiment of the presentinvention;

FIG. 7 is an exemplary user interface that may be used for entering dataassociated with a model, in connection with an embodiment of the presentinvention;

FIGS. 8-13 are tables illustrating model data/meta data generated inaccordance with an exemplary implementation of the present invention;and

FIG. 14 is a diagram illustrating an exemplary system for carrying outthe methods of the present invention.

DETAILED DESCRIPTION

The systems and method described herein relate to predictive anddescriptive modeling systems. More specifically, the systems and methodspertain to the creation, storage, retrieval, and maintenance of data andmetadata in predictive and descriptive modeling systems. The systemcreates and maintains model metadata, model executions, and theirresulting model outputs. Methods for capturing, classifying, anddocumenting model inputs and outputs are also provided. The apparatussupports mapping physical or logical structures in a database system viaa system catalog to a model for the purpose of defining model inputs.These inputs can be used in a one-to-one mapping or as part of a definedusage context (e.g., a derived field such as an indicator or calculatedmetric) that may utilize multiple fields or even other mappings. Aflexible storage solution may also be provided, which eliminates theneed for structural database changes for the deployment of new orupdated models. This leads to significant savings of time and money.These structures also facilitate retrieval and ensure consistentapplication integration via a standard table-based interface.Additionally, the model instance may provide an audit trail includingthe user, server, server network address, system process identifier, andtimestamp for the execution. Outputs from a model execution are taggedwith the corresponding model instance identifier, which allows analyststo see the history of models and their scores over time withoutambiguity.

Aspects of the present invention provide for a centralized predictiveknowledge repository, which contains the sum of an enterprise'spredictive experience. Previously, this knowledge was tacit, existing inthe minds of employees or scattered about network drives in unstructureddocuments and computer code. Consistency and structure are provided byembodiments of the invention. In particular, regardless of the type ofpredictive model used, or the inputs or outputs of model, the modelmetadata and model outputs are stored and managed. Previously, ad-hocdatabase structures had to be built for new models. Among the otheradvantages to this structural consistency is that applications consumingthe model outputs have a standardized method of retrieval. No matter howthe underlying predictive model changes, the retrieval of outputsremains consistent. This is advantageous because it reduces developmenttime and deployment cost, and increases speed to market.

Some aspects of the present invention provide real-time operatingability, in terms of optimized score management processes, outputstructure and accessibility.

As a knowledge repository, the process starts when the modeler entersdata into an application via, for example, a web-based user interface.Once entered, model information is available to the enterprise andlinked to the outputs produced by each model. Information that may becaptured includes the predictive technique, the model author, and thedata used as inputs to the models.

Regardless of the modeler's inputs describing the predictive model,every new model is assigned a model identifier, or Model_Id, thatuniquely identifies the model. Models built for a related purpose arealso assigned a Model_Group_Id. Start and end dates determine apredictive model's lifetime. An identification strategy such as this oneis key to enabling effective consumption of the resulting model scoresand measuring effectiveness.

Every time the model runs, an instance identifier is created, called theModel_Instance_Id, which directly precedes the execution of the model. Acreation date-time is logged and a status field is set to “R” (running).A user can view the data at this time, observe that a particular modelis running, find out on what server it is running on, and view othercompleted instances to understand how long the model will take tofinish. If the model completes successfully, the instance record isupdated and the status field is reset to “C” (complete). A communicationmay be sent to interested parties upon completion of the modelexecution.

When a model successfully operates, its outputs are stored in theapplication and are retrievable using Model_Instance_Id as a key. Thisallows for analytic evaluation of a model's scores over time, andultimately its historical performance. Application layers (e.g., viewsor semantic layers) store the most recent scores in a format convenientto consuming software applications, which greatly improves theperformance of consuming applications, particularly when large datavolumes are involved.

FIG. 1 illustrates the context in which embodiments of the invention maybe used. In particular, the context is one in which modeling environment101 is functionally dependent on a database 102, both reading data fromand writing data back to the database 102. FIG. 2 illustrates componentsof an exemplary production model execution environment, with functionaldependencies noted. Thus, FIG. 2 expands on FIG. 1 by showing how aproduction modeling environment may consist of scheduling and storagecomponents, as well as an execution engine. Thus, FIG. 2 shows that themodel execution engine 201 may read and write to a database 202. It mayalso invoke models from the model storage environment 203, and receivenotifications from the model storage environment 203. The model storageenvironment 203 may call the scheduler 204 and receive notificationsfrom the same. The database 202 feeds data to consuming applications205.

FIG. 3A and FIG. 3B each illustrate the dependencies of the scheduler204 and the model storage system 203, which components are integral toan enterprise-class statistical modeling environment 101. In particular,FIG. 3A is a diagram illustrating a production grade scheduler 204 andits dependencies (i.e., model execution engine 201; job 301; andcalendar 302; some of which invoke schedule 303), in accordance with anexemplary embodiment. FIG. 3B is a diagram illustrating the componentsof model storage system 203 and their dependencies (modeling environment101; job 301; model execution engine 201; and model 304), in accordancewith an exemplary embodiment.

FIG. 3C is a diagram illustrating an overview of which components withinthe modeling environment are invoked in connection with the processesdescribed herein. This exemplary flow refers to the basic componentsshown in FIG. 3A and FIG. 3B, as well as other components. Basically,the scheduler 204 is invoked and contacts the model storage host 203,which obtains the model metadata. The model instance generator 305generates a model instance 306 which is used to track the execution ofthe model (model execution engine 201) and the result set 307 ofexecution. The result set 307 is stored in model score consumption mart308, where it can be used by downstream applications 205.

FIG. 3D is a more detailed flowchart further illustrating the processoutlined in FIG. 3C. In particular, FIG. 3C introduces twosolution-specific components, the model instance generator 305 and themodel score consumption mart 308 which are used in connection with anembodiment of the present invention. FIG. 3D illustrates the sequence inwhich these components are active in the process, in an exemplaryembodiment. With reference to FIG. 3D, upon the occurrence of a triggerevent, schedule 204 is invoked, in step 309. In step 310, the schedulercontacts the model storage host 203. In step 311, the model storage host203 access model metadata from the model score consumption mart 308. Instep 312, the model storage host 312 invokes the model instance script.In step 313, the model instance generator 306 generates the modelinstance and passes it to the model execution engine 201. In step 314,the model execution engine 201 runs the model. In step 315, the modelexecution engine 201 captures additional system metadata and inserts themodel instance identifier into the model score consumption mart 308. Instep 316, it is determined whether the model execution completedsuccessfully. If not, in step 317, notifications are generated and themodel instance metadata in the model score consumption mart 308 isupdated with failed terminal status. If so, in step 318, the generatedmodel results are inserted in the model score consumption mart 308output table. Also, the model instance metadata is updated withsuccessful terminal status. The model instance generator 305 and themodel score consumption mart 308 comprise the apparatus for executingpredictive and descriptive models, whose main features and componentsare described below.

Model Instance

The relationship between the statistical model and the application ofthe model to data is referred to herein as an “instance,” or “modelinstance.” FIG. 4A depicts an entity-relationship model of statisticalmodels and how they relate to their instances. Every run of a modelcreates an instance; one instance may be related to a variety ofanalytic units and outputs, and many instances may be created over amodel's service lifetime. FIG. 4B also illustrates anentity-relationship model but further describes how the statisticalmodel relates to data within a database. Each data element is associatedwith details and, because a data element may be involved with multiplemodels, data elements are associated with roles for a particular model,as described in more detail below.

FIG. 5A is a logical data model showing certain (i.e., primary, foreign,and natural) keys for the entity-relationship model of statisticalmodels and their executions. For simplicity, this illustration does notinclude the model data. A model is uniquely determined by its ModelIdentifier (Model Id 501). A Model Instance 306, on the other hand, isuniquely determined by the Model Id 501 in combination with a startdatetime, a Job Id, and an Execution Engine Id. Here, job refers to thebatch program running the model on the execution engine.

To facilitate querying of a particular model instance from the database,the surrogate key Model_Instance_Id 502 is created. It is designed insuch a way that all elements of the natural key (Model Id 501, StartDatetime, Job Id, and Execution Engine Id) may be extracted throughparsing the field itself, accomplished through an encoding based on thehexadecimal system.

Model Outputs

The purpose of running a predictive or descriptive statistical model,i.e., creating a Model Instance 306, is to generate outputs that in someway describe an analytic unit of interest. FIG. 4A shows how a ModelInstance 306 relates to its outputs. The instance may create many outputrecords, but each output record was created from one and only one ModelInstance 306. In the entity-relationship modeling context, a ModelInstance Unit Output 503 is represented, where “Unit” stands for aparticular subtype of Model Instance Output. Model Instance Unit Output503 is referred to herein, where abstract units are identified with theattribute “Unit ID” 504.

FIG. 5A shows the primary and foreign keys related to the Model InstanceUnit Output 503 relation. The Model Instance ID 502 is a component ofthe key, whereas other components necessary for uniqueness include theunit identifier (Unit ID) 504 and the type of output (Output Type Id505). FIG. 5B is a logical data model showing certain (i.e., primary,foreign, and natural) keys for the entity-relationship model describinghow the statistical model relates to data within a database.

Referring back to FIG. 5A, other contextual information includes EventID and Event Date. Models are run for a reason. Thus, it is assumed thatevery event type of interest has a corresponding unique identifier—anEvent Id. Because some events are recurring (e.g., an “event” may be amonthly scoring), the event date is an important part of the context. Anadditional contextual field, “Standard Period Id,” includes informationon the business relevant time period or frequency.

An attribute of interest in the Model Instance Unit Output 503 relationis the Model Output Value 506. This field contains the outputs of modelswhich in some way describe or make a prediction about the unit ofinterest (hence, the phrase “predictive and descriptive models”).

Model Data

Referring back to FIGS. 4A and 4B, the manner in which a model relatesto its data is illustrated. Multiple models may use a particular dataelement, implying a many-to-many relationship between the Model and theData Element entities. To remedy this, an associative entity calledModel Data Element is created. This entity serves a purpose—one model'spredictor may be another model's target of prediction. The Data ElementRole entity, functionally related to Model Data Element, indicates thecontext of the data element in a particular model.

Focusing on the data element, without the context of the model, is theData Element entity. An important non-key attribute of the Data Elementrelation is the Data Element Derived Indicator, which indicates whetheradditional transformations have been applied to database columns tocreate the data element. If this indicator is false (or 0), then thefield is a direct mapping from a column in a physical database to a dataelement that can be used in a predictive or descriptive model. If theindicator is true (or 1), then some transformation has been applied to acolumn or columns from the database. In the case that multiple variablesare involved, there is a one-to-many relationship between Data Elementand the relation Data Element Detail, which includes all the physicaldatabase columns used in the creation of the data element. The exactnature of the transformation is not currently specified.

FIG. 5A shows the primary and foreign keys related to the model datacomponent of the Model Score Consumption Mart. The Model relation hasone foreign key and unique identifier, Model Id 501, which is pairedwith the data element identifier, Data Element Id 507, to form theprimary key of the Model Data Element table 508. Referring to FIG. 5B,the foreign key within this relation, Data Element Role Id 509, is theunique primary key in the entity, Data Element Role, which providescategorical information about the nature of the data element in thecontext of the model. Every data element in the Model Data Element 508relation is also necessarily represented in the Data Element 510relation, with Data Element Id 507 as the unique primary key.

The primary key of Data Element 510, Data Element Id 507, is alsocontained in the relation Data Element Detail 511. Since multipledatabase columns can be used to create a data element, there is aone-to-many relationship here, yet Data Element Id 507 is foreign keyrather than a primary key in the Data Element Detail 511 relation. Thisis because the database column identifier Data Element Detail Id issufficient to ensure uniqueness and identifiability of all databasecolumns.

Model Metadata

In addition to the production aspects of this apparatus and method forexecuting predictive and descriptive models, the Model Score ConsumptionMart 308 in particular provides a way to document and store metadataabout models.

Referring to FIG. 4A, the Model entity 401 has a one to one relationshipbetween the Abstract Model entity 402, the Statistical Modeling Toolentity 403, and the Model Purpose 404 entity.

As shown in FIG. 5A, the primary keys in these entities are all foreignkeys in the Model entity 401. The data from the Abstract Model entity402 is meant to give the analyst an idea of the technique that thepredictive or descriptive model was based on. For example, a decisiontree-based model will have a different output score distribution than aregression model with continuous predictors; the Abstract Model entity402 is designed to provide a quick glimpse into the type of model inquestion. The Statistical Modeling Tool entity 403 provides informationabout what software was used to estimate the model. Because models arebuilt for many purposes, with descriptive and predictive as two genericcategories, the Model Purpose entity 404 is meant to answer the questionof why the model was built.

FIGS. 6A and 6B together show an exemplary physical database schematicof the application component of the apparatus. Such a database model isphysically instantiated in a production database that is accessible toconsuming applications 205.

To facilitate the entry of model metadata into the application, softwareapplications featuring user interfaces may be used. FIG. 7 is anexemplary user interface that may be used in connection with theapplication for entering model metadata.

The following provides an example of how the systems and methodsdescribed herein can be used in connection with a business processreferred to herein as OYSR. By way of background, the OYSR model maps anumerical score to customer households with an impending insurancepolicy renewal, where higher scores correspond to a higher likelihood ofa beneficial effect when the proactive activity related to the policy iscarried out by an agent. The OYSR model runs nightly, and customerhouseholds are scored by the model when an auto or property insurancepolicy within the household is near renewal.

In the company's predictive modeling environment, in this example, afirst iteration of the OYSR model has been running since 11/11/2011. On03/10/2012, the model is to be replaced with an update built using morerecent data. The below describes the implementation using the apparatusdescribed herein and a first run of the model. Note that, in thisexample, only features of the apparatus necessary to illustratefunctionality are described, and certain other metadata fields areomitted.

Before the First Execution

As future executions depend upon the independent entry in the Modeltable, its information is described first. This information is enteredusing a user interface, e.g., as in FIG. 7, prior to the first executionof the model. In Table 1 (shown in FIG. 8), note that the updated OYSRmodel has been assigned Model_Id=9, whereas the previous model editionhad been previously assigned Model_Id=2. On the other hand, both modelsfall under Model_Group_Id=2. Thus the history of the OYSR modelinginitiative may be traced back using this field. Previous to the firstOYSR model (Model_Id=2) being built, the Model Group information seen inTable 2 (shown in FIG. 9) would have had been filled out.

When a business configuration manager fills enters information about theOYSR model update (Model_Id=9), he sets the business effective dates sothat the new model begins on a desired future date, in this case03/10/2012.

The model has been built with a language that the Model Execution Engine201 can parse and process. This code is stored in the location specifiedby Model Storage Path (See FIG. 3B). This path also includes a schedulein which the model will run.

After the business effective start date of Mar. 10, 2012, stored in theModel entity (Table 2), the scheduler follows a previously definedschedule, GDW_SPSS_DLY, stored in the Model Storage Path and named inthe Model Instance entity 306 (see also Table 3, FIG. 10). In thisexample, on 03/10/2012, at 3:00 AM, the schedule is called and thescheduler is invoked, running the job GDW_SPSS_MDL_OYS. This jobcontacts the Model Storage Host, which collects metadata from the Modeltable and uses information in the repository to generate a ModelInstance Id 502, in this case the string,“20120310030017-370-AC18A82D-116724.” A secure encrypted copy of theModel_Instance_Id 502 is passed to the Model Execution Engine 201,serverID, and the Model Execution Engine 201 inserts a record into theModel Instance Table, as seen in Table 3. This includes the starttimestamp of the Model Execution Engine (CREATE_DTTM) as well as that ofthe initial database insert (START_DTTM). Since the execution has notcompleted, the field END_DTTM field is left null and the Status is setto the code “R,” for “running.” At this time, the Model Execution Engine201 runs the OYSR model code as stored in the Model Storage Path. TheOYSR model includes business logic that queries the database forcustomer households with policy renewals in the near future (45 days orless). The logic also includes retrieves data elements, e.g. customertenure, and uses these data elements in a mathematical equation tocreate a propensity score.

The scores themselves are stored in the Model Instance Household Outputentity and given MODEL_OUTPUT_TYPE_ID=1, as shown in Table 4, shown inFIG. 11. Scores of this type are associated with the effectiveness ofOYSR activities. In addition to the raw model score, a business friendlyscore is also given, with Model_Output_Type_Id=4. Thus, each of thesample households is associated with two rows instead of one. Finally,the Business Event for the OYSR activity is a policy renewal, and theBusiness Event date is defined to be the policy renewal date.

After all households are scored, the Model Execution Engine 201 writesthe final timestamp END_DTTM in the Model Instance table, as well asupdating the status to “C” for complete, as shown in Table 5, FIG. 12.Messages are sent out indicating a successful completion, and consumingapplications may now retrieve scores from the Model_Instance_Hsld_Outputtable, or one of the views in the application layer of the Model ScoreConsumption Mart 308.

The model will continue to run as defined by the schedule in the ModelStorage Host. Table 6 (FIG. 13) shows the Model Instance entity afterthe updated OYSR model has run multiple times.

Exemplary hardware and software employed by the systems are nowgenerally described with reference to FIG. 14. Database server(s) 1400may include a database services management application 1406 that managesstorage and retrieval of data from the database(s) 1401, 1402. Thedatabases may be relational databases; however, other dataorganizational structure may be used without departing from the scope ofthe present invention. One or more application server(s) 1403 are incommunication with the database server 800. The application server 1403communicates requests for data to the database server 1400. The databaseserver 1400 retrieves the requested data. The application server 1403may also send data to the database server for storage in the database(s)1401, 1402. The application server 1403 comprises one or more processors1404, computer readable storage media 1405 that store programs (computerreadable instructions) for execution by the processor(s), and aninterface 1407 between the processor(s) 1404 and computer readablestorage media 1405. The application server may store the computerprograms referred to herein.

To the extent data and information is communicated over the Internet,one or more Internet servers 808 may be employed. The Internet server1408 also comprises one or more processors 1409, computer readablestorage media 1411 that store programs (computer readable instructions)for execution by the processor(s) 1409, and an interface 1410 betweenthe processor(s) 1409 and computer readable storage media 1411. TheInternet server 1408 is employed to deliver content that can be accessedthrough the communications network, e.g., by end user 1412. When data isrequested through an application, such as an Internet browser, theInternet server 1408 receives and processes the request. The Internetserver 1408 sends the data or application requested along with userinterface instructions for displaying a user interface.

The computers referenced herein are specially programmed to perform thefunctionality described herein as performed by the software programs.

The non-transitory computer readable storage media may include volatileand non-volatile, removable and non-removable media implemented in anymethod or technology for storage of information such ascomputer-readable instructions, data structures, program modules, orother data. Computer readable storage media may include, but is notlimited to, RAM, ROM, Erasable Programmable ROM (EPROM), ElectricallyErasable Programmable ROM (EEPROM), flash memory or other solid statememory technology, CD-ROM, digital versatile disks (DVD), or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bythe computer system.

It will be appreciated by those skilled in the art that changes could bemade to the exemplary embodiments shown and described above withoutdeparting from the broad inventive concept thereof. It is understood,therefore, that this invention is not limited to the exemplaryembodiments shown and described, but it is intended to covermodifications within the spirit and scope of the present invention asdefined by the claims. For example, specific features of the exemplaryembodiments may or may not be part of the claimed invention and featuresof the disclosed embodiments may be combined. Unless specifically setforth herein, the terms “a”, “an” and “the” are not limited to oneelement but instead should be read as meaning “at least one”.

It is to be understood that at least some of the figures anddescriptions of the invention have been simplified to focus on elementsthat are relevant for a clear understanding of the invention, whileeliminating, for purposes of clarity, other elements that those ofordinary skill in the art will appreciate may also comprise a portion ofthe invention. However, because such elements are well known in the art,and because they do not necessarily facilitate a better understanding ofthe invention, a description of such elements is not provided herein.

Further, to the extent that the method does not rely on the particularorder of steps set forth herein, the particular order of the stepsshould not be construed as limitation on the claims. The claims directedto the method of the present invention should not be limited to theperformance of their steps in the order written, and one skilled in theart can readily appreciate that the steps may be varied and still remainwithin the spirit and scope of the present invention.

What is claimed is:
 1. A computer implemented method comprising: storingmodel metadata for each of a plurality of models, the model metadatacomprising a statistical analysis technique identifier and one or moremodel input data identifiers; receiving a model execution requestcomprising data identifying one of the plurality of models, a modelexecution start date and a model execution end date; commencing, on themodel execution start date, execution of the model associated with themodel execution request; storing outputs of the executed model in adatabase, the outputs being associated with a model instance identifier,the model instance identifier identifying a model instance, wherein themodel instance comprises data describing a relationship between themodel associated with the model execution request and application of themodel associated with the model execution request to model input data;information describing a context for execution of the model; and modeloutput type information; and retrieving the outputs, using the modelinstance identifier, for analysis.
 2. A non-transitory computer readablestorage medium having stored thereon computer executable instructionsthat, when executed on a computer, configure the computer to perform amethod comprising: storing model metadata for each of a plurality ofmodels, the model metadata comprising a statistical analysis techniqueidentifier and one or more model input data identifiers; receiving amodel execution request comprising data identifying one of the pluralityof models, a model execution start date and a model execution end date;commencing, on the model execution start date, execution of the modelassociated with the model execution request; storing outputs of theexecuted model in a database, the outputs being associated with a modelinstance identifier, the model instance identifier identifying a modelinstance, wherein the model instance comprises data describing arelationship between the model associated with the model executionrequest and application of the model associated with the model executionrequest to model input data; information describing a context forexecution of the model; and model output type information; andretrieving the outputs, using the model instance identifier, foranalysis.
 3. A system comprising: memory operable to store at least oneprogram; and at least one processor communicatively coupled to thememory, in which the at least one program, when executed by the at leastone processor, causes the at least one processor to perform a methodcomprising the steps of: storing model metadata for each of a pluralityof models, the model metadata comprising a statistical analysistechnique identifier and one or more model input data identifiers;receiving a model execution request comprising data identifying one ofthe plurality of models, a model execution start date and a modelexecution end date; commencing, on the model execution start date,execution of the model associated with the model execution request;storing outputs of the executed model in a database, the outputs beingassociated with a model instance identifier, the model instanceidentifier identifying a model instance, wherein the model instancecomprises data describing a relationship between the model associatedwith the model execution request and application of the model associatedwith the model execution request to model input data; informationdescribing a context for execution of the model; and model output typeinformation; and retrieving the outputs, using the model instanceidentifier, for analysis.